build,refactor: update development tooling and package configuration #614
Conversation
egparedes
force-pushed
the
update-dev-tooling
branch
from
07272f4 to
87fe070
Compare
egparedes
force-pushed
the
update-dev-tooling
branch
from
f26b688 to
fb8fe5f
Compare
egparedes
force-pushed
the
update-dev-tooling
branch
from
fb8fe5f to
6c48671
Compare
ci/docker/base.Dockerfile
Outdated
| @@ -82,4 +82,4 @@ RUN pyenv update && \ | |||
| ENV PATH="/root/.pyenv/shims:${PATH}" | |||
|
|
|||
| ARG CUPY_PACKAGE=cupy-cuda11x | |||
| RUN pip install --upgrade pip setuptools wheel tox clang-format ${CUPY_PACKAGE} | |||
| RUN pip install --upgrade pip setuptools wheel uv nox clang-format ${CUPY_PACKAGE} | |||
There was a problem hiding this comment.
| RUN pip install --upgrade pip setuptools wheel uv nox clang-format ${CUPY_PACKAGE} | |
| RUN pip install --upgrade uv nox |
Probably that's the only thing that is useful here, because later we create a venv, which will not contain the other stuff.
Therefore, we don't have cupy in the environment, therefore the gpu test in tests/py2fgen/test_cli.py fails. Not sure what's the best way to define the cupy package version and then install it with nox/uv.
There was a problem hiding this comment.
Some more cleanup can be applied to the Dockerfile, but as discussed with @egparedes it can be done after this PR is merged.
For example, the pyenv is probably not needed and these lines can be removed:
ENV PYENV_ROOT /root/.pyenv
ENV PATH="/root/.pyenv/bin:${PATH}"
RUN pyenv update && \
pyenv install ${PYVERSION} && \
echo 'eval "$(pyenv init -)"' >> /root/.bashrc && \
eval "$(pyenv init -)" && \
pyenv global ${PYVERSION}
ENV PATH="/root/.pyenv/shims:${PATH}"
Instead, we could just install uv with:
RUN curl -LsSf https://astral.sh/uv/install.sh | sh
ENV PATH="/root/.local/bin:${PATH}"
|
cscs-ci run default |
noxfile.py
Outdated
| @nox.session(python=["3.10", "3.11"]) | ||
| @nox.parametrize("subpackage", MODEL_SUBPACKAGE_PATHS) | ||
| def test_model_stencils(session: nox.Session, subpackage: ModelSubpackagePath) -> None: | ||
| if subpackage != "common": # TODO: Enable tests |
There was a problem hiding this comment.
If I understand correctly, the stencil_tests from common were not run in main. Not sure if on purpose or by mistake.
|
cscs-ci run default |
ci/base.yml
Outdated
| @@ -81,6 +81,7 @@ build_image_aarch64: | |||
| variables: | |||
| CSCS_ADDITIONAL_MOUNTS: '["/project/d121/icon4py/ci/testdata:$TEST_DATA_PATH"]' | |||
| HPC_SDK_PATH: "/opt/nvidia/hpc_sdk/Linux_${ARCH}/22.11" | |||
| ICON4PY_NOX_UV_EXTRA_ARGS: "--extra=cuda11" | |||
There was a problem hiding this comment.
True, the cupy-cuda python package installed in the system environment in Docker build is not available in the uv environment. We can remove it from the Docker file, I guess.
|
cscs-ci run default |
|
cscs-ci run dace |
| elif subpackage == "common": | ||
| session.skip(f"tests broken") # TODO: Enable tests |
There was a problem hiding this comment.
here is the proof that they are failing in main if enabled https://gitlab.com/cscs-ci/ci-testing/webhook-ci/mirrors/5125340235196978/2255149825504669/-/jobs/8694767649
|
cscs-ci run default |
|
Mandatory Tests Please make sure you run these tests via comment before you merge!
Optional Tests To run benchmarks you can use:
To run tests and benchmarks with the DaCe backend you can use:
In case your change might affect downstream icon-exclaim, please consider running
For more detailed information please look at CI in the EXCLAIM universe. |
|
cscs-ci run default |
|
cscs-ci run dace |
This PR moves the development tooling to use modern and more capable tools like
uvandnox. It also adds a newicon4pyvirtual package in the root folder (used to gather all the model components) and moves out of theicon4py.model.commonsubpackage to the newicon4py.model.testingsubpackage the utilities only needed for testing purposes.uvtool is used to manage the dependencies of all subprojects thanks to its support for theworkspaceconcept which fits perfectly the monorepo structure of this repository. All requirements files have been replaced by dependency groups in the rootpyproject.tomlused byuv.noxis used to manage the testing environments instead oftox. Feature-wise is similar totoxbut it has a more user-friendly configuration using Python code instead of configuration files.